Refactor save_to_hub, and add option to push training checkpoints to Hub#1629
Refactor save_to_hub, and add option to push training checkpoints to Hub#1629NimaBoscarino wants to merge 8 commits intohuggingface:masterfrom
Conversation
|
@osanseviero Yup! Here's an example repo: https://huggingface.co/NimaBoscarino/STPushToHub-test |
|
@osanseviero I refactored it and now have a HubParameters class to hold the params! Here's a repo where I tested it all out: https://huggingface.co/NimaBoscarino/July25Test/commits/main If you think this looks good, I can write up some docstrings for it and I can add it to the Hugging Face section of the docs. |
osanseviero
left a comment
osanseviero
left a comment
There was a problem hiding this comment.
Thanks a lot! This is looking quite good, I left some final comments
osanseviero
left a comment
osanseviero
left a comment
There was a problem hiding this comment.
This looks good to me!
- Feel free to close discussions a resolved if they are resolved :)
- I left a comment about
local_repo_pathcreating temporary directories
APart from that, this looks in good state. Pinging @nreimers 🤗 to see what they think
|
Hello! This PR has been superseded by #2380. Also, Repository is now outdated. I do want to thank you for this time, and I'm sorry that it was never merged - I recognize that you (all) put time and effort into this.
|
3 participants
This PR contributes two things:
save_to_hubmethod has been refactored and simplified to use one of the new utility functions from the Hugging Face Hub client library, which uses thecreate_commitAPI..fit()to push checkpoints to the Hub.Pushing the checkpoints halts training (and there are no progress bars for the upload), which might not be ideal UX. Would it be better if the upload happened in the background while training continued?